Did solve memleak but this current mutex + dict implementation is complex and not very reliable
Instead handle this in the loader function of the iteration : prepare file-> table mapping in advance
And clean up when we don't need them using reference counting

That or remove the parallism completely from the parquet reading and instead do a simpler loader using the precomputed pieces to know what slices to read
Worth benchmarking a few solutions for the parquet alone

It might also be valuable to take into account down the line parallelization such as torch dataset workers. Maybe sequential parquet reading would be fine in that case.

done in another pr